Image processing device, endoscope apparatus, isolated point noise correction method, and information storage device

ABSTRACT

An image processing device includes an isolated point noise detection section, and an isolated point noise correction section. The isolated point noise detection section determines whether or not isolated point noise is included within a given area based on a first index value that represents the range of first to nth pixel values being obtained by arranging the pixel values of pixels within the given area including an attention pixel in ascending or descending order, and a second index value that represents the range of a pixel value group being obtained by excluding at least one of the first pixel value and the nth pixel value from the first to nth pixel values. The isolated point noise detection section determines whether or not the attention pixel is a pixel that corresponds to isolated point noise when it has been determined that isolated point noise is included within the given area.

CROSS REFERENCE TO RELATED APPLICATION

This application is a continuation of International Patent ApplicationNo. PCT/JP2013/061643, having an international filing date of Apr. 19,2013, which designated the United States, the entirety of which isincorporated herein by reference. Japanese Patent Application No.2012-137481 filed on Jun. 19, 2012 is also incorporated herein byreference in its entirety.

BACKGROUND

The present invention relates to an image processing device, anendoscope apparatus, an isolated point noise correction method, aninformation storage device, and the like.

Isolated point noise having a high noise level may occur due to a pixeldefect of an image sensor, or a transmission error that occurs whentransmitting the captured image via wireless transmission, for example.Such isolated point noise has been normally corrected by performing amedian filtering process on the image.

JP-A-2007-143120 discloses a method that determines that an attentionpixel is isolated point noise when the difference between the pixelvalue of the attention pixel and the pixel value of its peripheral pixelis equal to or larger than a first threshold value, and the differencebetween the pixel values of the peripheral pixels is equal to or smallerthan a second threshold value, and corrects the pixel value of theattention pixel that has been determined to be isolated point noisebased on the pixel value of the peripheral pixel.

SUMMARY

According to one aspect of the invention, there is provided an imageprocessing device comprising:

an isolated point noise detection section that detects isolated pointnoise that is included in an image; and

an isolated point noise correction section that corrects the isolatedpoint noise detected by the isolated point noise detection section,

the isolated point noise detection section including an isolated pointnoise presence/absence determination section and an attention pixelisolated point noise determination section,

the isolated point noise presence/absence determination sectiondetermining whether or not the isolated point noise is included within agiven area based on a first index value that represents a range of firstto nth pixel values, and a second index value that represents a range ofa pixel value group, the given area including an attention pixel that isan isolated point noise detection target, the first to nth pixel valuesbeing obtained by arranging pixel values of pixels within the given areain ascending or descending order, and the pixel value group beingobtained by excluding at least one of the first pixel value and the nthpixel value from the first to nth pixel values, and

the attention pixel isolated point noise determination sectiondetermining whether or not the attention pixel is a pixel thatcorresponds to the isolated point noise when it has been determined thatthe isolated point noise is included within the given area.

According to another aspect of the invention, there is provided an imageprocessing device comprising:

an isolated point noise detection section that detects isolated pointnoise that is included in an image; and

an isolated point noise correction section that corrects the isolatedpoint noise detected by the isolated point noise detection section,

the isolated point noise detection section determining whether or notthe isolated point noise is included within a given area including anattention pixel that is an isolated point noise detection target, basedon pixel values of pixels within the given area, and determining whetheror not the attention pixel is a pixel that corresponds to the isolatedpoint noise when it has been determined that the isolated point noise isincluded within the given area.

According to another aspect of the invention, there is provided anisolated point noise correction method comprising:

calculating a first index value that represents a range of first to nthpixel values, and a second index value that represents a range of apixel value group, the first to nth pixel values being obtained byarranging pixel values of pixels within a given area in ascending ordescending order, the pixel value group being obtained by excluding atleast one of the first pixel value and the nth pixel value from thefirst to nth pixel values, and the given area including an attentionpixel that is an isolated point noise detection target;

determining whether or not the isolated point noise is included withinthe given area based on the first index value and the second indexvalue;

determining whether or not the attention pixel is a pixel thatcorresponds to the isolated point noise when it has been determined thatthe isolated point noise is included within the given area; and

correcting a pixel value of the attention pixel when it has beendetermined that the attention pixel is a pixel that corresponds to theisolated point noise.

According to another aspect of the invention, there is provided acomputer-readable storage device with an executable image processingprogram stored thereon, wherein the image processing program instructs acomputer to perform steps of:

calculating a first index value that represents a range of first to nthpixel values, and a second index value that represents a range of apixel value group, the first to nth pixel values being obtained byarranging pixel values of pixels within a given area in ascending ordescending order, the pixel value group being obtained by excluding atleast one of the first pixel value and the nth pixel value from thefirst to nth pixel values, and the given area including an attentionpixel that is an isolated point noise detection target;

determining whether or not the isolated point noise is included withinthe given area based on the first index value and the second indexvalue;

determining whether or not the attention pixel is a pixel thatcorresponds to the isolated point noise when it has been determined thatthe isolated point noise is included within the given area; and

correcting a pixel value of the attention pixel when it has beendetermined that the attention pixel is a pixel that corresponds to theisolated point noise.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a configuration example of an image processingdevice.

FIG. 2 illustrates a detailed configuration example of an isolated pointnoise correction processing section according to a first embodiment.

FIGS. 3A to 3D are views illustrating a process performed by acorrection area extraction section.

FIG. 4 is a view illustrating a first dynamic range and a second dynamicrange.

FIG. 5 is a view illustrating a process performed by an isolated pointnoise determination section and a process performed by an attentionpixel isolated point noise determination section.

FIG. 6 is a view illustrating a process performed by an isolated pointnoise determination section and a process performed by an attentionpixel isolated point noise determination section.

FIGS. 7A to 7D are views illustrating a process performed by an isolatedpoint noise determination section and a process performed by anattention pixel isolated point noise determination section.

FIG. 8 is a flowchart illustrating an isolated point noise correctionprocess.

FIG. 9 is a view illustrating a highlight correction process.

FIG. 10 illustrates a detailed configuration example of an isolatedpoint noise correction processing section according to a secondembodiment.

FIG. 11 illustrates a detailed configuration example of a highlightcorrection processing section.

FIG. 12 is a view illustrating a process performed by a first highlightdetermination area extraction section and a process performed by anattention pixel highlight determination section.

FIG. 13 is a view illustrating a process performed by a second highlightdetermination area extraction section and a process performed by anadjacent pixel highlight determination section.

FIG. 14 is a flowchart illustrating a highlight correction process.

FIG. 15 illustrates a configuration example of an endoscope apparatus.

DESCRIPTION OF EXEMPLARY EMBODIMENTS

According to one embodiment of the invention, there is provided an imageprocessing device comprising:

an isolated point noise detection section that detects isolated pointnoise that is included in an image; and

an isolated point noise correction section that corrects the isolatedpoint noise detected by the isolated point noise detection section,

the isolated point noise detection section including an isolated pointnoise presence/absence determination section and an attention pixelisolated point noise determination section,

the isolated point noise presence/absence determination sectiondetermining whether or not the isolated point noise is included within agiven area based on a first index value that represents a range of firstto nth pixel values, and a second index value that represents a range ofa pixel value group, the given area including an attention pixel that isan isolated point noise detection target, the first to nth pixel valuesbeing obtained by arranging pixel values of pixels within the given areain ascending or descending order, and the pixel value group beingobtained by excluding at least one of the first pixel value and the nthpixel value from the first to nth pixel values, and

the attention pixel isolated point noise determination sectiondetermining whether or not the attention pixel is a pixel thatcorresponds to the isolated point noise when it has been determined thatthe isolated point noise is included within the given area.

According to one embodiment of the invention, whether or not isolatedpoint noise is included within the given area including the attentionpixel is determined based on the first index value and the second indexvalue, and whether or not the attention pixel is a pixel thatcorresponds to isolated point noise is determined when it has beendetermined that isolated point noise is included within the given area.This makes it possible to adaptively detect isolated point noisecorresponding to a local change in pixel value of the image.

Exemplary embodiments of the invention are described below. Note thatthe following exemplary embodiments do not in any way limit the scope ofthe invention laid out in the claims. Note also that all of the elementsdescribed below in connection with the following exemplary embodimentsshould not necessarily be taken as essential elements of the invention.

1. Outline

Random noise, fixed pattern noise, and isolated point noise have beenknown as noise that may be included in image data. The term “randomnoise” refers to noise that occurs due to a photoelectric conversionprocess performed by an image sensor. The term “fixed pattern noise”refers to noise that occurs due to a variation in circuitcharacteristics (e.g., amplifier gain) provided to each readout linewhen reading a pixel value from an image sensor using a multi-linereadout process. The term “isolated point noise” refers to noise havinga high noise level that occurs due to a pixel defect of an image sensor,or a transmission error that occurs when transmitting the captured imagevia wireless transmission, for example.

Random noise and fixed pattern noise have been normally reduced byperforming a process that utilizes a smoothing filter on an attentionpixel and peripheral pixels adjacent thereto. In recent years, anedge-preserving smoothing filter (e.g., c-filter or bilateral filter)that can reduce noise while preserving an edge has been used as thesmoothing filter.

However, since isolated point noise is regarded as an edge when removingisolated point noise, it is impossible to reduce isolated point noiseusing the edge-preserving smoothing filter. Isolated point noise isnormally corrected by performing a median filtering process on a givenextracted area (attention pixel and its peripheral pixels). The medianfilter has frequency characteristics that allow an edge component toremain as compared with a non-edge-preserving smoothing filter (e.g.,Gaussian filter), and can remove the largest pixel value or the smallestpixel value within the extracted area (processing target) thatcorresponds to isolated point noise.

However, a decrease in resolution occurs when the median filter isuniformly applied to the entire image. Therefore, a method is normallyemployed that detects isolated point noise by performing an isolatedpoint noise correction process, and performs a smoothing filteringprocess (e.g., median filtering process) on only the detected isolatedpoint noise and its peripheral pixels. In particular, since the positionof isolated point noise rarely changes when the isolated point noise hasoccurred due to a defect pixel of the image sensor, it is possible todetect the defect pixel (white defect pixel or black defect pixel) bycapturing a given chart in advance. The position of the defect pixel canbe recorded on a memory provided in an imaging device, and thecorrection process can be performed only on the position of the defectpixel recorded in the memory.

However, when isolated point noise has occurred due to a random event(e.g., an error during image transmission), it is necessary to detectthe isolated point noise that has occurred randomly from the transmittedimage, and correct the detected isolated point noise. For example,JP-A-2007-143120 discloses the following method that successivelydetects isolated point noise. Specifically, the attention pixel isdetermined to be isolated point noise when the difference between thepixel value of the attention pixel and the pixel value of its peripheralpixel is equal to or larger than a first threshold value, and thedifference between the pixel values of the peripheral pixels is equal toor smaller than a second threshold value.

According to this method, it is possible to detect isolated point noisewith high accuracy when the area around the attention pixel is flat.However, the detection accuracy decreases when an edge is present in thearea around the attention pixel. This is because the first thresholdvalue and the second threshold value are fixed, and are not synchronizedwith a local change in pixel value of the processing target image.

According to several embodiments of the invention, whether or notisolated point noise is present at a pixel among pixels P(−1, −1) toP(1, 1) within a given area including an attention pixel is determined(described later with reference to FIG. 4 and the like). When it hasbeen determined that isolated point noise is present at a pixel amongthe pixels P(−1, −1) to P(1, 1) within the given area, whether or notthe attention pixel P(0, 0) is the isolated point noise is determined.When it has been determined that the attention pixel P(0, 0) is theisolated point noise, the isolated point noise is corrected.

According to this configuration, it is possible to determine whether ornot a prominent pixel value that corresponds to isolated point noise ispresent within the given area. Therefore, even when a local change inpixel value (e.g., edge) is present around the attention pixel, it ispossible to determine whether or not isolated point noise is presentwithin the given area without being affected by such a local change inpixel value. When it has been determined that a prominent pixel value ispresent within the given area, whether or not the pixel is the attentionpixel is determined to detect isolated point noise.

2. First Embodiment 2.1. Image Processing Device

A first embodiment of the invention is described in detail below. FIG. 1illustrates a configuration example of an image processing deviceaccording to the first embodiment.

The image processing device illustrated in FIG. 1 includes an imagingsection 100, an OB (optical black) processing section 110, an isolatedpoint noise correction processing section 120, a random noise reductionprocessing section 130, a WB (white balance) processing section 140, ademosaicing processing section 150, a CMS (color management system)processing section 160, a gamma processing section 170, an edgeenhancement processing section 180, an output section 190, and a controlsection 200. Note that the configuration of the image processing deviceis not limited to the configuration illustrated in FIG. 1. Variousmodifications may be made, such as omitting some (e.g., imaging section100, control section 200, and output section 190) of the elements, oradding other elements.

The control section 200 is bidirectionally connected to the OBprocessing section 110, the isolated point noise correction processingsection 120, the random noise reduction processing section 130, the WBprocessing section 140, the demosaicing processing section 150, the CMSprocessing section 160, the gamma processing section 170, and the edgeenhancement processing section 180, and exchanges parameters used duringthe process performed by each section with each section.

The imaging section 100 includes an imaging lens, an image sensor, andan A/D conversion section, for example. The image sensor is implementedby a CCD image sensor or a CMOS image sensor, for example. In the firstembodiment, the image sensor is a single-chip image sensor having aprimary-color Bayer array. Note that the image sensor is not limitedthereto. Various other image sensors may also be used. An image formedon the image sensor through the imaging lens is photoelectricallyconverted corresponding to each pixel of the image sensor to generate ananalog signal. The analog signal is converted into a digital signal bythe A/D conversion section, and the digital signal (image signal) isoutput to the OB processing section 110.

The OB processing section 110 determines the zero level of the imagesignal based on the image signal obtained from the light-blocking areaof the image sensor, subtracts the zero level from the image signalobtained from the exposure area of the image sensor, and outputs thezero level-corrected image signal (hereinafter appropriately referred toas “image”) to the isolated point noise correction processing section120.

The isolated point noise correction processing section 120 performs anisolated point noise detection process and an isolated point noisecorrection process on the input image, and outputs the image subjectedto the isolated point noise correction process to the random noisereduction processing section 130. The details of the isolated pointnoise detection process and the isolated point noise correction processare described later.

The random noise reduction processing section 130 performs anedge-preserving adaptive noise filtering process and the like on theinput image to reduce random noise, and outputs the image (from whichrandom noise has been reduced) to the WB processing section 140.

The WB processing section 140 performs a white balance correctionprocess that multiplies the R signal and the B signal of the input imageby a white balance coefficient, and outputs the image subjected to thewhite balance correction process to the demosaicing processing section150.

The demosaicing processing section 150 performs an interpolation processon the input Bayer-array image to generate a color image (color imagesignal), and outputs the color image to the CMS processing section 160.For example, the interpolation process interpolates the pixel value(signal value) of the processing target attention pixel using the pixelvalue of its peripheral pixels. The term “color image” used hereinrefers to an image in which each pixel has an R (red) pixel value, a G(green) pixel value, and a B (blue) pixel value.

The CMS processing section 160 performs a color conversion process onthe input color image so that the display image has the desired color,and outputs the color image subjected to the color conversion process tothe gamma processing section 170.

The gamma processing section 170 performs a gamma conversion process onthe input color image corresponding to the grayscale characteristics ofthe display device, and outputs the color image subjected to the gammaconversion process to the edge enhancement processing section 180.

The edge enhancement processing section 180 performs an edge enhancementprocess that extracts an edge component from the input color image,multiplies the edge component by the desired gain, and adds theresulting edge component to the color image, and outputs the color imagesubjected to the edge enhancement process to the output section 190.

The output section 190 is a liquid crystal display or an organic ELdisplay, for example. The output section 190 displays the input colorimage to the user so that the color image can be observed as imageinformation.

2.2. Isolated Point Noise Correction Processing Section

FIG. 2 illustrates a detailed configuration example of the isolatedpoint noise correction processing section 120. The isolated point noisecorrection processing section 120 includes a correction area extractionsection 121, an identical-color signal selection section 122, anisolated point noise correction section 125 (isolated point pixel noisecorrection processing section), and an isolated point noise detectionsection 126. The point noise detection section 126 includes an isolatedpoint noise presence/absence determination section 123 (isolated pointnoise area detection section) and an attention pixel isolated pointnoise determination section 124.

The position and the size of the extracted area for correcting isolatedpoint noise is supplied to the correction area extraction section 121from the control section 200 as the process parameters. A firstcoefficient Co1 for determining whether or not an isolated point noisepixel is included within a pixel area that corresponds to an identicalcolor and has been selected by the identical-color signal selectionsection 122 is supplied to the isolated point noise presence/absencedetermination section 123 from the control section 200 as the processparameter. When the isolated point noise presence/absence determinationsection 123 has determined that an isolated point noise pixel isincluded within the pixel area that corresponds to an identical color, asecond coefficient Co2 for determining whether or not the pixel value ofthe attention pixel (i.e., a pixel situated at the center of the pixelarea) within the pixel area that corresponds to an identical color isisolated point noise is supplied to the isolated point noisepresence/absence determination section 123 from the control section 200as the process parameter.

The image output from the OB processing section 110 is input to thecorrection area extraction section 121. The correction area extractionsection 121 extracts the attention pixel and a given N×M (N and M arenatural numbers) pixel area situated around the attention pixel(processing target area) based on the process parameter from the controlsection 200. Note that an example in which the N×M pixel area is a 5×5pixel area (see FIG. 3A) is described below. FIG. 3A illustrates anexample in which the pixel value of the upper left pixel of the N×Mpixel area is the R signal. Note that the pixel value of the upper leftpixel of the N×M pixel area may be the Gr signal, the B signal, or theGb signal. The G signal is divided into the Gr signal and the Gb signalfor convenience of the process performed by the isolated point noisepresence/absence determination section 123, the process performed by theattention pixel isolated point noise determination section 124, and theprocess performed by the isolated point noise correction section 125.Note that the configuration is not limited thereto. The Gr signal andthe Gb signal may collectively be handled as the G signal, and theprocess may be performed using a number of pixels differing from that ofthe R signal and the B signal.

The N×M pixel area extracted by the correction area extraction section121 is input to the identical-color signal selection section 122. Theidentical-color signal selection section 122 selects the pixel values ofthe pixels that correspond to a color identical to that of the attentionpixel from the N×M pixel area, and outputs the selected pixel values tothe isolated point noise presence/absence determination section 123. Theidentical-color signal selection section 122 outputs the pixel value ofthe attention pixel to the isolated point noise correction section 125.For example, when the pixel value of the attention pixel is an R signalvalue, the R signal values of 3×3 pixels included within the N×M pixelarea are output (see FIG. 3A). When the pixel value of the attentionpixel is a Gr signal value, the Gr signal values of 3×3 pixels areoutput (see FIG. 3B). When the pixel value of the attention pixel is aGb signal value, the Gb signal values of 3×3 pixels are output (see FIG.3C). When the pixel value of the attention pixel is a B signal value,the B signal values of 3×3 pixels are output (see FIG. 3D).

As illustrated in FIG. 4, the isolated point noise presence/absencedetermination section 123 one-dimensionally sorts (arranges) the pixelvalues of the 3×3 pixels that correspond to an identical color inascending order, and stores the pixel values in a memory (notillustrated in the drawings). In FIG. 4, S1 is the smallest value amongthe pixel values that correspond to an identical color, S2 is the secondsmallest value that is larger than the smallest value S1, Mid is amiddle value, L1 is the largest value among the pixel values thatcorrespond to an identical color, and L2 is the second largest valuethat is smaller than the largest value L1. Although an example in whichthe pixel values are sorted in ascending order is described below, thepixel values may be sorted in descending order. Although an example inwhich the middle value is a median value is described below, the middlevalue may be an arbitrary value between the value S2 and the value L2.Note that the median value refers to the center value among the pixelvalues that are arranged in ascending or descending order. For example,the median value refers to the fifth value among nine pixel values thatare arranged in ascending or descending order.

When only one isolated point noise is included within the 3×3 pixels,the smallest value S1 or the largest value L1 is a candidate for theisolated point noise. In order to determine whether the smallest valueS1 or the largest value L1 is the isolated point noise, the isolatedpoint noise presence/absence determination section 123 calculates thedifference between the largest value L1 and the smallest value S1 as afirst dynamic range dRange1 (first index value), and calculates thedifference between the value L2 and the value S2 as a second dynamicrange dRange1 (second index value). The first dynamic range DRange1 isan index value that represents the range from the smallest value S1 tothe largest value L1 among the pixel values within the 3×3 pixel localarea, and the second dynamic range DRange2 is an index value thatrepresents the range from the value S2 (smallest value) to the value L2(largest value) among the pixel values within the 3×3 pixel local areaexcluding the largest value L1 and the smallest value S1.

In FIG. 5, an edge EG is present within the 3×3 pixel local area. Ifonly one pixel value that is significantly larger or smaller than theremaining pixel values is present, the one pixel value is considered tobe isolated point noise. Therefore, when isolated point noise is notpresent within the 3×3 pixel local area, at least two pixels among the3×3 pixels have a large pixel value at an almost equal level (see thewhite pixels in FIG. 5), and at least two pixels among the 3×3 pixelshave a small pixel value at an almost equal level (see the hatchedpixels in FIG. 5). In this case, the difference between the firstdynamic range DRange1 and the second dynamic range DRange2 is small (seeFIG. 7B). In FIG. 7B, the difference between the middle value Mid andthe value L4 represents the difference in pixel value due to the edge.When no edge is present within the 3×3 pixel local area (i.e., flat), nodifference in pixel value is observed, and the difference between thefirst dynamic range DRange1 and the second dynamic range DRange2 issmall.

Therefore, the isolated point noise presence/absence determinationsection 123 compares the first dynamic range DRange1 with the seconddynamic range DRange2 that is multiplied by the coefficient Co1 from thecontrol section 200 (i.e., the value Co1×dRange2), and determines thatthe largest value L1 or the smallest value S1 is isolated point noisewhen the first dynamic range DRange1 is larger than the valueCo1×dRange2. Note that the first dynamic range DRange1 and the seconddynamic range DRange2 may be compared using another method. For example,a value obtained by subtracting the second dynamic range dRange2 fromthe first dynamic range dRange1 may be compared with a given thresholdvalue to determine whether or not the largest value L1 or the smallestvalue S1 is isolated point noise.

The isolated point noise presence/absence determination section 123outputs the determination result (isolated point noise presence/absencedetermination information) as to whether or not a pixel value among thepixel values of the 3×3 pixels that correspond to an identical color isisolated point noise, to the attention pixel isolated point noisedetermination section 124. When the isolated point noisepresence/absence determination section 123 has determined that a pixelvalue among the pixel values of the 3×3 pixels that correspond to anidentical color is isolated point noise, the isolated point noisepresence/absence determination section 123 outputs the middle value Mid,the second dynamic range dRange2, and the pixel value of the attentionpixel to the attention pixel isolated point noise determination section124.

The attention pixel isolated point noise determination section 124determines whether or not the attention pixel is isolated point noisebased on the middle value Mid, the second dynamic range dRange2, and thepixel value of the attention pixel when the isolated point noisepresence/absence determination information represents that isolatedpoint noise is present. Specifically, the attention pixel isolated pointnoise determination section 124 determines that the attention pixel isisolated point noise when the absolute difference value between themiddle value Mid and the pixel value of the attention pixel P(0, 0)(pixel value P(0, 0)) (i.e., value |Mid−P(0, 0)|) is larger than athreshold value (see FIG. 7C). Note that the pixel value of theattention pixel P(0, 0) may be referred to as “pixel value P(0, 0)”.This similarly applies to other pixels. The threshold value is a valueobtained by multiplying the second dynamic range dRange2 by the secondcoefficient Cot supplied from the control section 200 (i.e., valueCo2×dRange2).

When the value |Mid−P(0, 0)| is larger than the value Co2×dRange2, it isdetermined that the pixel value of the attention pixel P(0, 0) is largerthan the second dynamic range DRange2 that represents the range of thepixel values within the 3×3 pixel local area excluding the largest valueL1 and the smallest value S1. Therefore, when the value |Mid−P(0, 0)| islarger than the value Co2×dRange2, it is likely that the attention pixelhas the largest value L1 or the smallest value S1, and it may bedetermined that the attention pixel is isolated point noise. Theattention pixel isolated point noise determination section 124 outputsan isolated point noise correction flag (determination result) and themiddle value Mid to the isolated point noise correction section 125. Theisolated point noise correction flag is set to “ON” (correction) (e.g.,logic level “1”) when it has been determined that the attention pixel isisolated point noise. The isolated point noise correction flag is set to“OFF” (no correction) (e.g., logic level “0”) when it has beendetermined that the attention pixel is not isolated point noise.

Note that the edge EG may be present between one corner pixel among the3×3 pixels and the remaining eight pixels (see FIG. 6). In this case,since only the pixel value of the one corner pixel is significantlylarger or smaller than the remaining pixel values, the isolated pointnoise presence/absence determination section 123 determines that the onecorner pixel is isolated point noise. However, since the value |Mid-P(0,0)1 is smaller than the value Co2×dRange2 (see FIG. 7D), the attentionpixel isolated point noise determination section 124 determines that theattention pixel is not isolated point noise, and isolated point noise isnot detected. Specifically, even when apparent isolated point noiseoccurs within 3×3 pixels due to the edge EG, it is possible to determinewhether or not the apparent isolated point noise is the actual isolatedpoint noise.

The pixel value of the attention pixel output from the identical-colorsignal selection section 122, and the middle value Mid and the isolatedpoint noise correction flag output from the attention pixel isolatedpoint noise determination section 124 are input to the isolated pointnoise correction section 125. The isolated point noise correctionsection 125 outputs the middle value Mid as the pixel value of theattention pixel when the isolated point noise correction flag is set to“ON”, and outputs the pixel value of the attention pixel directly whenthe isolated point noise correction flag is set to “OFF”. The isolatedpoint noise correction process on the attention pixel is thus completed.Note that the pixel value of the attention pixel need not necessarily becorrected using the middle value Mid. The pixel value of the attentionpixel may be corrected using a value obtained from the pixel values S1to L1 included within the 3×3 pixel local area.

2.3. Details of Isolated Point Noise Correction Process

The isolated point noise correction process is described in detail belowwith reference to FIGS. 7A to 7D. FIGS. 7A to 7D are schematic views inwhich the pixel values included within the 3×3 pixel local area (seeFIG. 5) are arranged in ascending order from the left. The first dynamicrange dRange1 is the difference between the pixel value L1 and the pixelvalue S1, and the second dynamic range dRange2 is the difference betweenthe pixel value L2 and the pixel value S2. A change in pixel valuewithin the local area is represented using the first dynamic rangedRange1 and the second dynamic range dRange2.

FIG. 7A is a schematic view when one isolated point noise is presentwithin the local area, and FIG. 7B is a schematic view when no isolatedpoint noise is present within the local area. The hatched pixel valuesrepresent a pixel value that is relatively smaller than isolated pointnoise, and the pixel value that is not hatched represents a large pixelvalue that corresponds to isolated point noise.

When an edge (e.g., the edge EG in FIG. 5) is present within the localarea, the hatched pixel values are divided into two groups. In FIG. 7A,eight pixel values S1 to L2 are divided into a first group (S1 to Mid)and a second group (L4 to L2). In FIG. 7B, nine pixel value S1 to L1 aredivided into a first group (S1 to Mid) and a second group (L4 to L1). Asillustrated in FIG. 7B, the difference between the first dynamic rangedRange1 and the second dynamic range dRange2 is small when isolatedpoint noise is not present. Specifically, since the first dynamic rangedRange1 and the second dynamic range dRange2 change adaptively withrespect to the gap in pixel value (i.e., the difference between thefirst group (S1 to Mid) and the second group (L4 to L1)) due to theedge, the first dynamic range dRange1 and the second dynamic rangedRange2 are not affected by the edge, and are almost equal. Asillustrated in FIG. 7A, the difference between the first dynamic rangedRange1 and the second dynamic range dRange2 is large when one isolatedpoint noise is present. Specifically, it is possible to determinewhether or not isolated point noise is present within the local areawithout being affected by the presence of an edge by detecting thedifference between the first dynamic range dRange1 and the seconddynamic range dRange2.

Although an example in which whether or not one isolated point noise ispresent within the local area is determined using the first dynamicrange dRange1 and the second dynamic range dRange2 (=L2−S2) has beendescribed above, the configuration is not limited thereto. For example,the difference between the pixel value L1 and the pixel value S2, andthe difference between the pixel value L2 and the pixel value S1 may becalculated, and the difference between the pixel value L1 and the pixelvalue S2, or the difference between the pixel value L2 and the pixelvalue S1, whichever is smaller, may be used as the second dynamic rangedRange2. Alternatively, whether or not two or more isolated point noisesare present within the local area may be determined. For example,whether or not two or more isolated point noises are present within thelocal area may be determined using the first dynamic range dRange1 and athird dynamic range dRange3 (=L3−S3). Note that it is possible to reducethe probability that an edge that is present in the periphery of thelocal area is erroneously determined to be isolated point noise bydetermining whether or not one isolated point noise is present withinthe local area using the first dynamic range dRange1 and the seconddynamic range dRange2.

FIG. 7C is a schematic view when the attention pixel is isolated pointnoise, and FIG. 7D is a schematic view when the attention pixel is notisolated point noise. Whether or not the attention pixel is isolatedpoint noise is determined by determining whether or not the absolutedifference value |Mid−P(0, 0)| between the middle value Mid (medianvalue) among the nine pixel values S1 to L1 and the pixel value of theattention pixel P(0, 0) is within the second dynamic range dRange2.

Specifically, when the attention pixel (pixel value: P(0, 0)) is notisolated point noise, the absolute difference value |Mid−P(0, 0)| iswithin the second dynamic range dRange2 (see FIG. 7D). This is becausethe second dynamic range dRange2 represents the difference in pixelvalue due to an edge present within the local area, and the absolutedifference value between the middle value Mid and the pixel value of theattention pixel P(0, 0) within the second dynamic range dRange2 issmaller than the second dynamic range dRange2. Since the second dynamicrange dRange2 changes adaptively with respect to an edge, the aboverelationship is satisfied regardless of the presence or absence of anedge. When the attention pixel (pixel value P(0, 0)) is isolated pointnoise, the absolute difference value |Mid−P(0, 0)| is larger than thesecond dynamic range dRange2. It is possible to determine whether or notthe attention pixel is isolated point noise by comparing the absolutedifference value |Mid−P(0, 0)| with the second dynamic range dRange2 asdescribed above. It is also possible to accurately detect whether or notisolated point noise is present at the attention pixel even when an edgeis present, by performing the two-step determination process describedabove with reference to FIGS. 7A to 7D.

FIG. 8 is a flowchart illustrating the isolated point noise correctionprocess performed by the isolated point noise correction processingsection 120. When the isolated point noise correction process hasstarted, the correction area extraction section 121 extracts the pixelvalues within the local area (i.e., the attention pixel and the pixelssituated around the attention pixel (N×M pixels)) (i.e., processingtarget area) from the input image, and the identical-color signalselection section 122 selects the pixel values (3×3 pixels) of thepixels that correspond to a color identical to that of the attentionpixel from the pixel values within the local area (step S21).

The isolated point noise presence/absence determination section 123detects the largest value L1 and the second largest value L2 from theselected pixel values (step S22). The isolated point noisepresence/absence determination section 123 detects the smallest value S1and the second smallest value S2 from the selected pixel values (stepS23). The isolated point noise presence/absence determination section123 subtracts the smallest value S1 from the largest value L1 tocalculate the first dynamic range dRange1 (step S24). The isolated pointnoise presence/absence determination section 123 subtracts the secondsmallest value S2 from the second largest value L2 to calculate thesecond dynamic range dRange2 (step S25). The isolated point noisepresence/absence determination section 123 multiplies the second dynamicrange dRange2 by the coefficient Co1 set by the control section 200 tocalculate a dynamic range threshold value Th1 (step S26).

The isolated point noise presence/absence determination section 123determines whether or not the first dynamic range dRange1 is larger thanthe threshold value Th1 (step S27). When the isolated point noisepresence/absence determination section 123 has determined that the firstdynamic range dRange1 is equal to or smaller than the threshold valueTh1, a step S32 is performed. When the isolated point noisepresence/absence determination section 123 has determined that the firstdynamic range dRange1 is larger than the threshold value Th1, theisolated point noise presence/absence determination section 123 detectsthe middle value Mid (step S28).

The attention pixel isolated point noise determination section 124multiplies the second dynamic range dRange2 by the coefficient Co2 setby the control section 200 to calculate an isolated point noisedetermination threshold value Th2 (step S29). The attention pixelisolated point noise determination section 124 determines whether or notthe absolute difference value |Mid−P(0, 0)| between the middle value Midand the pixel value of the attention pixel P(0, 0) is larger than thethreshold value Th2 (step S30). When the attention pixel isolated pointnoise determination section 124 has determined that the absolutedifference value |Mid−P(0, 0)| is equal to or smaller than the thresholdvalue Th2, the step S32 is performed. When the attention pixel isolatedpoint noise determination section 124 has determined that the absolutedifference value |Mid−P(0, 0)| is larger than the threshold value Th2,the attention pixel isolated point noise determination section 124replaces the pixel value of the attention pixel P(0, 0) with the middlevalue Mid (step S31).

Whether or not all of the processing target pixels, have been processedis determined (step S32). When it has been determined that all of theprocessing target pixels have not been processed, the step S21 isperformed again. When it has been determined that all of the processingtarget pixels have been processed, the process is terminated.

Although an example in which the isolated point noise correction processis performed on the image captured by the image sensor having aprimary-color Bayer array has been described above, the configuration isnot limited thereto. Specifically, the isolated point noise correctionprocess can also be performed on an image captured by a primary-colorthree-chip image sensor, a complementary-color single-ship image sensor,or a frame-sequential monochrome single-chip image sensor, as long as itis possible to define the local area that consists of the attentionpixel and the pixels that are situated around the attention pixel andcorrespond to a color identical to that of the attention pixel.

According to the first embodiment, the image processing device includesthe isolated point noise detection section 126 that detects isolatedpoint noise that is included in an image, and the isolated point noisecorrection section 125 that corrects the isolated point noise detectedby the isolated point noise detection section 126 (see FIG. 2). Theisolated point noise detection section 126 includes the isolated pointnoise presence/absence determination section 123 and the attention pixelisolated point noise determination section 124. As described above withreference to FIG. 4 and the like, the isolated point noisepresence/absence determination section 123 determines whether or not theisolated point noise is included within a given area (local area) basedon a first index value dRange1 (first dynamic range) that represents therange of first to nth pixel values S1 to L1, and a second index valuedRange2 (second dynamic range) that represents the range of a pixelvalue group S2 to L2. The first to nth pixel values S1 to L1 areobtained by arranging (sorting) the pixel values of the pixels P(−1, −1)to P(1, 1) within the given area in ascending or descending order, thegiven area including the attention pixel P(0, 0) that is the isolatedpoint noise detection target. The pixel value group S2 to L2 is obtainedby excluding at least one of the first pixel value S1 and the nth pixelvalue L1 from the first to nth pixel values S1 to L1. The attentionpixel isolated point noise determination section 124 determines whetheror not the attention pixel P(0, 0) is a pixel that corresponds to theisolated point noise when it has been determined that the isolated pointnoise is included within the given area.

According to this configuration, it is possible to accurately detectwhether or not the isolated point noise is present within the given areaeven when the image shows a local change in pixel value (e.g., an imageof an edge area), by calculating the first index value dRange1 and thesecond index value dRange2 corresponding to the pixels P(−1, −1) toP(1, 1) within the given area. Specifically, since the first index valuedRange1 and the second index value dRange2 change adaptively withrespect to a local change in pixel value (as described above withreference to FIG. 7A and the like), the isolated point noise can bedetected without being affected by a local change in pixel value. Sincewhether or not the attention pixel is the isolated point noise isdetermined only when the isolated point noise has been detected withinthe local area, the isolated point noise detection accuracy with respectto the attention pixel can be improved.

Note that the index value (first index value and second index value)that represents the pixel value range is an index value that representsthe range of the pixel value distribution, and increases as the range ofthe pixel value distribution increases. For example, the index valueincreases as the largest value or the smallest value of the pixel valuedistribution increases. The index value increases in a local area thatincludes an edge since the range of the pixel value distributionincreases as compared with a flat local area that does not include anedge. Specifically, the index value changes adaptively with respect to achange in the range of the local pixel value distribution due to an edgeor the like.

The isolated point noise presence/absence determination section 123 maycalculate the difference |L1−S1| between the first pixel value S1 andthe nth pixel value L1 as the first index value dRange1, and calculatethe difference |L2−S2| between the largest value and the smallest valueamong the pixel values S2 to L2 included in the pixel value group as thesecond index value dRange2.

According to this configuration, it is possible to calculate the firstindex value dRange1 that represents the range of the first to nth pixelvalues S1 to L1, and the second index value dRange2 that represents therange of the pixel values S2 to L2 included in the pixel value group.Since a gap in pixel value due to an edge is included between thelargest value and the smallest value, the first index value dRange1 andthe second index value dRange2 change adaptively corresponding to thepresence or absence of an edge.

The pixel value group may include the second to (n−1)th pixel values S2to L2 among the first to nth pixel values L1 to S1. The isolated pointnoise presence/absence determination section 123 may calculate thedifference |L2−S2| between the second pixel value S2 and the (n−1)thpixel value L2 included in the pixel value group as the second indexvalue dRange2.

According to this configuration, it is possible to detect isolated pointnoise irrespective of whether the isolated point noise is white noise(high-brightness noise) or black noise (low-brightness noise). Note thatthe pixel value group may consist of the first to (n−1)th pixel valuesS1 to L2, and the isolated point noise presence/absence determinationsection 123 may calculate the second index value dRange2=|L2−S1|. Inthis case, since the pixel value L1 is the largest value when the pixelvalues are arranged in ascending order, it is possible to detect whiteisolated point noise. Alternatively, the pixel value group may consistof the second to nth pixel values S2 to L1, and the isolated point noisepresence/absence determination section 123 may calculate the secondindex value dRange2=|L1−S2|. In this case, since the pixel value S1 isthe smallest value when the pixel values are arranged in ascendingorder, it is possible to detect black isolated point noise. The pixelvalue group may be obtained by excluding two or more pixel values on thelargest value side (or the smallest value side) from the first to nthpixel values S1 to L1.

3. Second Embodiment 3.1. Isolated Point Noise Correction ProcessingSection

In FIG. 9, a highlight having a given size is formed on a primary-colorsingle-chip image sensor. When the isolated point noise correctionprocess is performed in such a case, the pixels of the highlight may beerroneously determined to be isolated point noise, and may beovercorrected.

For example, five adjacent pixels are included in the highlight area inwhich brightness saturation occurs. In the example illustrated in FIG.9, the five pixels included in the highlight area correspond to two Rsignals, one Gr signal, one Gb signal, and one B signal. The isolatedpoint noise correction process described above in connection with thefirst embodiment detects isolated point noise when only one pixel valuethat is significantly larger or smaller than the remaining pixel valuesis present within the 3×3 pixel local area. Therefore, when the isolatedpoint noise correction process is performed on the five pixels includedin the highlight area, the R signals are not detected as isolated pointnoise, and the Gr signal, the Gb signal, and the B signal are detectedas isolated point noise. In this case, only the R signals remainuncorrected, and the white highlight area is reproduced as a red spotarea. In the second embodiment, a process that corrects suchovercorrection due to a highlight is performed.

FIG. 10 illustrates a detailed configuration example of an isolatedpoint noise correction processing section 120 according to the secondembodiment. The isolated point noise correction processing section 120illustrated in FIG. 10 includes a correction area extraction section121, an identical-color signal selection section 122, an isolated pointnoise correction section 125 (isolated point noise pixel correctionprocessing section), an isolated point noise detection section 126, anda highlight correction processing section 801 (highlight detectioncorrection processing section and overcorrection correction processingsection). The isolated point noise detection section 126 includes anisolated point noise presence/absence determination section 123(isolated point noise area detection section), and an attention pixelisolated point noise determination section 124. Note that the sameelements as those described above in connection with the firstembodiment are respectively indicated by the same reference signs, anddescription thereof is appropriately omitted.

An example in which the image sensor is a single-chip image sensorhaving a primary-color Bayer array. Note that the configuration is notlimited thereto. Various other image sensors may also be used.

The isolated point noise correction flag output from the attention pixelisolated point noise determination section 124, the isolated point noisecorrection result output from the isolated point noise correctionsection 125, the white balance coefficient output from the controlsection 200, and the image (that is not subjected to the isolated pointnoise correction process) output from the OB processing section 110, areinput to the highlight correction processing section 801. The highlightcorrection processing section 801 determines whether or not the pixelfor which the isolated point noise correction flag is set to “ON” (i.e.,the pixel that has been subjected to the isolated point noise correctionprocess) is a pixel within the highlight area. When it has beendetermined that the pixel for which the isolated point noise correctionflag is set to “ON” is a pixel within the highlight area, the highlightcorrection processing section 801 outputs the original pixel valueinstead of the corrected pixel value.

3.2. Highlight Correction Processing Section

FIG. 11 illustrates a detailed configuration example of the highlightcorrection processing section 801. The highlight correction processingsection 801 includes a first highlight determination area extractionsection 901 (white balance correction processing section), a secondhighlight determination area extraction section 902, an attention pixelhighlight determination section 903, an adjacent pixel highlightdetermination section 904, an attention pixel highlight decision section905, an attention pixel correction processing section 906, and anisolated point noise correction flag storage section 907.

The isolated point noise correction flag output from the attention pixelisolated point noise determination section 124 is input to the isolatedpoint noise correction flag storage section 907. The isolated pointnoise correction flags corresponding to the number of necessary linesare stored in the isolated point noise correction flag storage section907 until they correspond to the area extracted by the first highlightdetermination area extraction section 901 and the area extracted by thesecond highlight determination area extraction section 902.

The image that is output from the OB processing section 110 and has notbeen subjected to the isolated point noise correction process, theisolated point noise correction flag output from the isolated pointnoise correction flag storage section 907, and the white balancecoefficient output from the control section 200, are input to the firsthighlight determination area extraction section 901. The image that hasnot been subjected to the isolated point noise correction process isstored in a line memory (not illustrated in the drawings).

As illustrated in FIG. 12, the first highlight determination areaextraction section 901 extracts a given two-dimensional area (e.g., 3×3pixel area) from the image stored in the line memory, the giventwo-dimensional area being formed around the position of the pixel forwhich the isolated point noise correction flag is set to “ON”. The areaextracted by the first highlight determination area extraction section901 is referred to as “first highlight determination area”, and thecenter pixel thereof is referred to as “first highlight determinationattention pixel”.

The first highlight determination area extraction section 901 (whitebalance correction processing section) performs a white balancecorrection process that multiplies the R signal and the B signal withinthe first highlight determination area by the white balance coefficient,and outputs the pixel values obtained by the white balance correctionprocess to the attention pixel highlight determination section 903. Thewhite balance correction process is performed since the highlight isoriginally white, and the RGB pixel values are equal. The determinationprocess described later determines whether or not the pixel is ahighlight taking account of the correlation between the pixel valuesthat differ in color. The determination process is affected when thewhite balance is not adjusted (i.e., the pixel values that differ incolor may not be equal). Therefore, the RGB gain is adjusted byperforming the white balance correction process.

The pixel values within the first highlight determination area (thathave been subjected to the white balance correction process) output fromthe first highlight determination area extraction section 901, a firstcoefficient (that differs from the first coefficient Co1 described abovein connection with the first embodiment) output from the control section200, and the isolated point noise correction flag (corresponding to eachpixel within the first highlight determination area) output from theisolated point noise correction flag storage section 907, are input tothe attention pixel highlight determination section 903.

As illustrated in FIG. 12, the attention pixel highlight determinationsection 903 extracts the pixel value of a pixel for which the isolatedpoint noise correction flag is set to “OFF” from pixels NP1 to NP4(i.e., adjacent pixels that differ in color) that are adjacent to thefirst highlight determination attention pixel in the upward/downwarddirection and the rightward/leftward direction (vertical scan directiony and horizontal scan direction x). In FIG. 12, the isolated point noisecorrection flag is represented by “(ON)” or “(OFF)”. The attention pixelhighlight determination section 903 calculates the absolute differencevalue between the extracted pixel value and the pixel value of the firsthighlight determination attention pixel. When a plurality of absolutedifference values have been calculated, the attention pixel highlightdetermination section 903 selects the smallest value from the pluralityof absolute difference values. When the isolated point noise correctionflag is set to “ON” for all of the adjacent pixels NP1 to NP4, theattention pixel highlight determination section 903 sets the absolutedifference value to the largest value so that the pixels NP1 to NP4 arenot determined to be a highlight.

The attention pixel highlight determination section 903 divides thepixel value of the first highlight determination attention pixel by thefirst coefficient to calculate a threshold value (attention pixelhighlight determination threshold value), and compares the selectedabsolute difference value with the threshold value. The attention pixelhighlight determination section 903 determines that the first highlightdetermination attention pixel is a highlight candidate when the selectedabsolute difference value is smaller than the threshold value, anddetermines that the first highlight determination attention pixel is nota highlight candidate when the selected absolute difference value isequal to or larger than the threshold value. The attention pixelhighlight determination section 903 outputs the determination result tothe attention pixel highlight decision section 905 as an attention pixelhighlight determination flag. The attention pixel highlightdetermination flag is set to “ON” (e.g., logic level “1”) when it hasbeen determined that the first highlight determination attention pixelis a highlight candidate, and is set to “OFF” (e.g., logic level “0”)when it has been determined that the first highlight determinationattention pixel is not a highlight candidate. The attention pixelhighlight determination section 903 outputs information about thedirection of the adjacent pixel corresponding to the selected absolutedifference value with respect to the first highlight determinationattention pixel to the adjacent pixel highlight determination section904 as information about the correlation direction. The correlationdirection is the upward direction (−y-direction in FIG. 13), thedownward direction (+y-direction in FIG. 13), the leftward direction(−x-direction in FIG. 13), or the rightward direction (+x-direction inFIG. 13). For example, when the adjacent pixel NP1 in FIG. 12corresponds to the selected absolute difference value, the correlationdirection is the leftward direction (−x-direction). Note that thecorrelation direction is hereinafter appropriately referred to as“highlight correlation direction”.

The image (that has been subjected to the isolated point noisecorrection process) output from the isolated point noise correctionprocess part 125, and the isolated point noise correction flag outputfrom the isolated point noise correction flag storage section 907, areinput to the second highlight determination area extraction section 902.The image that has been subjected to the isolated point noise correctionprocess is stored in a line memory (not illustrated in the drawings).

As illustrated in FIG. 13, the second highlight determination areaextraction section 902 extracts a given two-dimensional area (e.g., 5×5pixel area) from the image stored in the line memory, the giventwo-dimensional area being formed around the position of the pixel forwhich the isolated point noise correction flag is set to “ON”. The areaextracted by the second highlight determination area extraction section902 is referred to as “second highlight determination area”, and thecenter pixel thereof is referred to as “second highlight determinationattention pixel”. The second highlight determination area extractionsection 902 outputs the pixel values within the extracted secondhighlight determination area to the adjacent pixel highlightdetermination section 904.

The pixel values within the second highlight determination area outputfrom the second highlight determination area extraction section 902, asecond coefficient (that differs from the second coefficient Co2described above in connection with the first embodiment) output from thecontrol section 200, and information about the highlight correlationdirection output from the attention pixel highlight determinationsection 903, are input to the adjacent pixel highlight determinationsection 904. Note that an example in which the highlight correlationdirection is the leftward direction (−x-direction in FIG. 13) isdescribed below.

As illustrated in FIG. 13, the adjacent pixel highlight determinationsection 904 extracts the pixel value of the pixel NP1 that is adjacentto the second highlight determination attention pixel in the highlightcorrelation direction (−x-direction) from the second highlightdetermination area. The adjacent pixel highlight determination section904 also extracts the pixel values of pixels SC1 to SC4 that areidentical in color with the pixel NP1 and situated closest to the pixelNP1 in the direction y that is orthogonal to the highlight correlationdirection from the second highlight determination area. The adjacentpixel highlight determination section 904 determines whether or not thepixel values of the adjacent pixels NP1 and NP2 have a correlation withthe pixel values of the closest pixels SC1 to SC4, and determines thatthe adjacent pixel is a highlight when the pixel values of the adjacentpixels NP1 and NP2 do have a correlation with the pixel values of theclosest pixels SC1 to SC4.

Specifically, the adjacent pixel highlight determination section 904averages the pixel values of two pixels among the closest pixels thatare situated at a short distance to calculate two average pixel values.Specifically, the adjacent pixel highlight determination section 904calculates the average value (first average value) of the pixel valuesof the closest pixels SC1 and SC2, and the average value (second averagevalue) of the pixel values of the closest pixels SC3 and SC4. Theadjacent pixel highlight determination section 904 multiplies each ofthe first average pixel value and the second average pixel value by thesecond coefficient output from the control section 200 to calculate afirst threshold value and a second threshold value (adjacent pixelhighlight determination first threshold value and adjacent pixelhighlight determination second threshold value) that respectivelycorrespond to the first average pixel value and the second average pixelvalue. The adjacent pixel highlight determination section 904 calculatesthe absolute difference value between the pixel value of the adjacentpixel NP1 and the first average value, and compares the absolutedifference value with the first threshold value. The adjacent pixelhighlight determination section 904 calculates the absolute differencevalue between the pixel value of the adjacent pixel NP1 and the secondaverage value, and compares the absolute difference value with thesecond threshold value.

The adjacent pixel highlight determination section 904 determines thatat least one (i.e., the pixel NP1) of the pixels NP1 to NP4 adjacent tothe second highlight determination attention pixel is a highlight whenat least one absolute difference value is larger than the correspondingthreshold value. Specifically, the adjacent pixel highlightdetermination section 904 determines that the pixel NP1 is a highlightat least when the absolute difference value between the pixel value ofthe pixel NP1 and the first average value is larger than the firstthreshold value, or when the absolute difference value between the pixelvalue of the pixel NP1 and the second average value is larger than thesecond threshold value. The adjacent pixel highlight determinationsection 904 outputs the determination result to the attention pixelhighlight decision section 905 as an adjacent pixel highlightdetermination flag. The adjacent pixel highlight determination flag isset to “ON” (e.g., logic level “1”) when it has been determined that thepixel NP1 is a highlight, and is set to “OFF” (e.g., logic level “0”)when it has been determined that the pixel NP 1 is not a highlight.

The attention pixel highlight decision section 905 determines that theattention pixel is a highlight when both the attention pixel highlightdetermination flag and the adjacent pixel highlight determination flagare set to “ON”, and determines that the attention pixel is not ahighlight when at least one of the attention pixel highlightdetermination flag and the adjacent pixel highlight determination flagis set to “OFF”. Specifically, the attention pixel highlightdetermination section 903 calculates a logical AND of the attentionpixel highlight determination flag and the adjacent pixel highlightdetermination flag, and outputs the calculation result to the attentionpixel correction processing section 906 as a highlight determinationflag. The highlight determination flag is set to “ON” when both theattention pixel highlight determination flag and the adjacent pixelhighlight determination flag are set to “ON”, and is set to “OFF” whenat least one of the attention pixel highlight determination flag and theadjacent pixel highlight determination flag is set to “OFF”.

The attention pixel highlight decision section 905 calculates a NAND ofthe attention pixel highlight determination flag and the adjacent pixelhighlight determination flag, and outputs the calculation result to theisolated point noise correction flag storage section 907 as a revisedisolated point noise correction flag. The revised isolated point noisecorrection flag is set to “OFF” when the highlight determination flag isset to “ON”, and is set to “ON” when the highlight determination flag isset to “OFF”. The isolated point noise correction flag storage section907 updates the isolated point noise correction flag recordedcorresponding to the pixel position with the revised isolated pointnoise correction flag corresponding to the pixel position. Specifically,when a pixel that has been detected as isolated point noise has beendetermined to be a highlight, the isolated point noise correction flagcorresponding to this pixel is set to “OFF” from “ON”.

The attention pixel correction processing section 906 outputs the pixelvalue of the first highlight determination attention pixel before theisolated point noise correction process (i.e., the pixel value beforethe white balance correction process is performed by the first highlightdetermination area extraction section 901) instead of the pixel value ofthe second highlight determination attention pixel. The attention pixelcorrection processing section 906 outputs the pixel value of the secondhighlight determination attention pixel when the input highlightdetermination flag is set to “OFF” (i.e., when it has been determinedthat at least one of the attention pixel and the adjacent pixel is not ahighlight).

In the example illustrated in FIG. 13, the Gr signal corresponding tothe attention pixel is determined to be isolated point noise, and the Rsignals corresponding to the adjacent pixels NP1 and NP2 are notdetermined to be isolated point noise for the reason described abovewith reference to FIG. 9. Therefore, a red false color occurs in thehighlight area when the isolated point noise correction process isperformed.

According to the second embodiment, the pixel value of the attentionpixel that has been overcorrected as isolated point noise can bereturned to the original pixel value (i.e., the pixel value beforecorrection). Specifically, since the attention pixel and the adjacentpixels NP1 and NP2 are included within the highlight area, and have asimilar pixel value, the attention pixel highlight determination section903 determines that the attention pixel is a highlight candidate. Sincethe adjacent pixels NP1 and NP2 included within the highlight area havea large pixel value, the difference from the pixel values of the closestpixels SC1 to SC4 is large, and the adjacent pixel highlightdetermination section 904 determines that the adjacent pixels NP1 andNP2 are a highlight. When it has been determined that the attentionpixel is a highlight candidate, and the adjacent pixels NP1 and NP2 area highlight, the attention pixel highlight decision section 905determines that the attention pixel is a highlight, and the attentionpixel correction processing section 906 returns the pixel value of theattention pixel to the original pixel value (i.e., the pixel valuebefore correction). This makes it possible to correct overcorrection dueto a highlight.

3.3. Highlight Correction Process

FIG. 14 is a flowchart illustrating the process performed by thehighlight correction processing section 801. The first highlightdetermination area extraction section 901 determines whether or not theisolated point noise correction flag recorded in the isolated pointnoise correction flag storage section 907 is set to “ON” (i.e., whetheror not the attention pixel has been subjected to the isolated pointnoise correction process) (step S51). When the isolated point noisecorrection flag corresponding to the attention pixel is set to “OFF”,the next pixel is set to be the attention pixel, and the step S51 isperformed. When the isolated point noise correction flag correspondingto the attention pixel is set to “ON”, an area that has a given size(e.g., 3×3 pixels) and is formed around the position of the attentionpixel is extracted from the image that is not subjected to the isolatedpoint noise correction process (see FIG. 12) (step S52).

The first highlight determination area extraction section 901 performsthe white balance correction process on the pixel values within theextracted area (step S53). The attention pixel highlight determinationsection 903 selects a pixel (e.g., the pixel NP1) among the pixels NP1to NP4 that are situated adjacent to the first highlight detectionattention pixel in the extracted area for which the isolated point noisecorrection flag is set to “OFF”, and of which the pixel value after thewhite balance correction process has the maximum correlation with thefirst highlight detection attention pixel (step S54). The attentionpixel highlight determination section 903 determines whether or not thefirst highlight detection attention pixel is a highlight candidate basedon the pixel value of the selected pixel and the pixel value of thefirst highlight detection attention pixel (steps S55 and S56). When ithas been determined that the first highlight detection attention pixelis not a highlight candidate, the next pixel is set to be the attentionpixel, and the step S51 is performed.

When it has been determined that the first highlight detection attentionpixel is a highlight candidate, the second highlight determination areaextraction section 902 extracts a given area (e.g., 5×5 pixel area) fromthe image subjected to the isolated point noise correction process (seeFIG. 13) (step S57). The adjacent pixel highlight determination section904 determines whether or not the adjacent pixel (NP1) that has themaximum correlation with the second highlight detection attention pixelis a highlight based on the pixel values of the pixels SC1 to SC4 thatare identical in color with the second highlight detection attentionpixel, and are situated closest to the second highlight detectionattention pixel within the extracted area, and the pixel value of theadjacent pixel (NP1) selected by the attention pixel highlightdetermination section 903 (step S58).

The attention pixel highlight decision section 905 determines whether ornot the first highlight detection attention pixel has been determined tobe a highlight candidate, and the adjacent pixel (NP1) that has themaximum correlation with the second highlight detection attention pixelhas been determined to be a highlight, and determines whether or not thesecond highlight detection attention pixel is a highlight (step S59).When it has been determined that the second highlight detectionattention pixel is not a highlight candidate, the next pixel is set tobe the attention pixel, and the step S51 is performed. When it has beendetermined that the second highlight detection attention pixel is ahighlight, the attention pixel correction processing section 906 returnsthe pixel value of the second highlight detection attention pixel to thepixel value before the isolated point noise correction process isperformed, and the isolated point noise correction flag storage section907 sets (returns) the isolated point noise correction flagcorresponding to the second highlight detection attention pixel to “OFF”(step S60).

Whether or not all of the pixels within the image have been processed isthen determined (step S61). When all of the pixels within the image havenot been processed, the step S51 is performed. When all of the pixelswithin the image have been processed, the process is terminated.

According to the second embodiment, the image processing device includesthe highlight correction processing section 801 (see FIG. 10). Thehighlight correction processing section 801 determines whether or notthe attention pixel that has been corrected by the isolated point noisecorrection section 125 is a pixel within the highlight area, and returnsthe pixel value of the attention pixel to the pixel value before beingcorrected by the isolated point noise correction section 125 when it hasbeen determined the attention pixel is a pixel within the highlightarea.

According to this configuration, even when a pixel within the highlightarea has been overcorrected as isolated point noise, the overcorrectedpixel value can be returned to the pixel value before the isolated pointnoise correction process is performed. This makes it possible to improvethe isolated point noise correction accuracy, and suppress occurrence ofan artifact in the highlight area.

The image for which isolated point noise is corrected may be an image inwhich pixels that differ in color (e.g., RGB pixels) are disposed (e.g.,disposed in a Bayer array). The highlight correction processing section801 may include the attention pixel highlight determination section 903(see FIG. 11). The attention pixel highlight determination section 903may compare the pixel value of the attention pixel that has not beencorrected by the isolated point noise correction section 125 with thepixel values of different-color adjacent pixels NP1 to NP4 that differin color from the attention pixel and are situated adjacent to theattention pixel to determine whether or not the attention pixel is acandidate for a pixel within the highlight area.

According to this configuration, since the attention pixel and thedifferent-color adjacent pixels NP1 to NP4 have a small difference inpixel value when the attention pixel and the different-color adjacentpixels NP 1 to NP4 are included within an identical highlight area, itis possible to detect that it is likely that the attention pixel isincluded within the highlight area.

The attention pixel highlight determination section 903 may select apixel (e.g., pixel NP1) among the different-color adjacent pixels NP1 toNP4 that has a pixel value closest to the pixel value of the attentionpixel as a selected pixel (see FIG. 12). The highlight correctionprocessing section 801 may include the adjacent pixel highlightdetermination section 904 (see FIG. 11). When the attention pixelhighlight determination section 903 has determined that the attentionpixel is a candidate for a pixel within the highlight area, the adjacentpixel highlight determination section 904 may compare the pixel value ofthe selected pixel NP1 with the pixel values of the pixels SC1 to SC4that are identical in color with the selected pixel NP1 and are situatedat a given position with respect to the selected pixel NP1 to determinewhether or not the selected pixel NP1 is a pixel within the highlightarea.

For example, the given position with respect to the selected pixelrefers to the positions of four pixels (i.e., the pixels SC1 to SC4)among the pixels that are identical in color with the selected pixelthat are situated closest to the attention pixel in the direction(y-direction) that is orthogonal to the direction of the selected pixelwith respect to the attention pixel (i.e., the −x-direction in FIG. 13when the selected pixel is the pixel NP1).

Even when the attention pixel highlight determination section 903 hasdetermined that the attention pixel is a highlight candidate, theattention pixel may not be a highlight when the adjacent pixel is notincluded within the highlight area. According to the second embodiment,since it is possible to determine whether or not the adjacent pixel isincluded within the highlight area, it is possible to determine whetheror not the attention pixel is included within the highlight area.

4. Imaging Device

FIG. 15 illustrates a configuration example of an endoscope apparatus asa configuration example of an imaging device to which the above imageprocessing device is applied. The endoscope apparatus includes a lightsource section 500, an imaging section 510 (insertion section), acontrol device 520 (processing section), a display section 530, and anexternal OF section 540.

The light source section 500 includes a light source 501 that emitsillumination light, a light intensity control section 502 that controlsthe intensity of the illumination light, and a condenser lens 503 thatfocuses the illumination light emitted from the light source 501. Theimaging section 510 is formed to be elongated and flexible so that theimaging section 510 can be inserted into the observation target.

The imaging section 510 includes a light guide fiber 504 that guides theillumination light focused by the condenser lens 503, and anillumination lens 505 that diffuses the illumination light guided by thelight guide fiber 504 to illuminate the object. The imaging section 510also includes an objective lens 511 and a focus lens 513 that form anobject image, a focus lens driver section 512 that drives the focus lens513, an image sensor 515 that captures the object image, and an A/Dconversion section 516 that converts an analog signal from the imagesensor 515 into a digital signal.

The control device 520 includes an image processing section 521 (imageprocessing device) that performs image processing on the image inputfrom the A/D conversion section 516, a control section 522 that controlseach section of the endoscope apparatus, and a focus control section 523that performs a focus adjustment process by controlling the focus lensdriver section 512. The image processing section 521 performs theisolated point noise correction process, the highlight correctionprocess, and the like (see above).

Although an example in which the imaging device is an endoscopeapparatus has been described above, the embodiments of the invention arenot limited thereto. The embodiments of the invention may be applied tovarious imaging devices. For example, the imaging device may be adigital camera that captures a still image, a video camera that capturesa movie, a capsule endoscope, or the like. For example, a capsuleendoscope is configured so that the capsule situated inside the body ofthe subject and the receiver situated outside the body of the subjectcommunicate via wireless communication, and random isolated point noisemay occur due to a transmission error during wireless communication.According to the embodiments of the invention, it is possible toappropriately correct isolated point noise even when random isolatedpoint noise has occurred in the vicinity of an edge area.

Although the embodiments of the invention have been described abovetaking an example in which each section of the image processing deviceis implemented by hardware, the embodiments of the invention are notlimited thereto. For example, the embodiments of the invention may alsobe applied to the case where a known computer system (e.g., work stationor personal computer) is used as the image processing device. In thiscase, each section of the image processing device may be implemented byproviding a program (image processing program) that implements theprocess performed by each section of the image processing section inadvance, and causing the CPU of the computer system to execute the imageprocessing program.

More specifically, a program stored in an information storage device isread, and executed by a processor (e.g., CPU). The information storagedevice (computer-readable device) stores a program, data, and the like.The information storage device may be an arbitrary recording device thatrecords a program that can be read by a computer system, such as aportable physical device (e.g., CD-ROM, USB memory, MO disk, DVD disk,flexible disk (FD), magnetooptical disk, or IC card), a stationaryphysical device (e.g., HDD, RAM, or ROM) that is provided inside oroutside a computer system, or a communication device that temporarilystores a program during transmission (e.g., public line connectedthrough a modem, or a local area network or a wide area network to whichanother computer system or a server is connected).

Specifically, a program is recorded in the recording device so that theprogram can be read by a computer. A computer system (i.e., a devicethat includes an operation section, a processing section, a storagesection, and an output section) implements the image processing deviceby reading the program from the recording device, and executing theprogram. Note that the program need not necessarily be executed by acomputer system. The embodiments of the invention may similarly beapplied to the case where another computer system or a server executesthe program, or another computer system and a server execute the programin cooperation.

The image processing device and the like according to the embodiments ofthe invention may include a processor and a memory. The processor may bea central processing unit (CPU), for example. Note that the processor isnot limited to a CPU. Various other processors such as a graphicsprocessing unit (GPU) or a digital signal processor (DSP) may also beused. The processor may be a hardware circuit that includes an ASIC. Thememory stores a computer-readable instruction. Each section of the imageprocessing device and the like according to the embodiments of theinvention is implemented by causing the processor to execute theinstruction. The memory may be a semiconductor memory (e.g., SRAM orDRAM), a register, a hard disk, or the like. The instruction may be aninstruction included in an instruction set of a program, or may be aninstruction that causes a hardware circuit of the processor to operate.

Although only some embodiments of the invention and the modificationsthereof have been described in detail above, those skilled in the artwill readily appreciate that many modifications are possible in theembodiments and the modifications thereof without materially departingfrom the novel teachings and advantages of the invention. A plurality ofelements described in connection with the above embodiments and themodifications thereof may be appropriately combined to implement variousconfigurations. For example, some of the elements described inconnection with the above embodiments and the modifications thereof maybe omitted. Arbitrary elements among the elements described inconnection with different embodiments and the modifications thereof maybe appropriately combined. Specifically, various modifications andapplications are possible without materially departing from the novelteachings and advantages of the invention. Any term cited with adifferent term having a broader meaning or the same meaning at leastonce in the specification and the drawings can be replaced by thedifferent term in any place in the specification and the drawings.

What is claimed is:
 1. An image processing device comprising: anisolated point noise detection section that detects isolated point noisethat is included in an image; and an isolated point noise correctionsection that corrects the isolated point noise detected by the isolatedpoint noise detection section, the isolated point noise detectionsection including an isolated point noise presence/absence determinationsection and an attention pixel isolated point noise determinationsection, the isolated point noise presence/absence determination sectiondetermining whether or not the isolated point noise is included within agiven area based on a first index value that represents a range of firstto nth pixel values, and a second index value that represents a range ofa pixel value group, the given area including an attention pixel that isan isolated point noise detection target, the first to nth pixel valuesbeing obtained by arranging pixel values of pixels within the given areain ascending or descending order, and the pixel value group beingobtained by excluding at least one of the first pixel value and the nthpixel value from the first to nth pixel values, and the attention pixelisolated point noise determination section determining whether or notthe attention pixel is a pixel that corresponds to the isolated pointnoise when it has been determined that the isolated point noise isincluded within the given area.
 2. The image processing device asdefined in claim 1, the isolated point noise presence/absencedetermination section calculating a difference between the first pixelvalue and the nth pixel value as the first index value, and calculatinga difference between a largest value and a smallest value among pixelvalues included in the pixel value group as the second index value. 3.The image processing device as defined in claim 2, the pixel value groupincluding second to (n−1)th pixel values among the first to nth pixelvalues, and the isolated point noise presence/absence determinationsection calculating a difference between the second pixel value and the(n−1)th pixel value included in the pixel value group as the secondindex value.
 4. The image processing device as defined in claim 1, theisolated point noise presence/absence determination section determiningthat the isolated point noise is included within the given area when thefirst index value is larger than a value obtained by multiplying thesecond index value by a first coefficient.
 5. The image processingdevice as defined in claim 1, the attention pixel isolated point noisedetermination section determining that the attention pixel is a pixelthat corresponds to the isolated point noise when an absolute differencevalue between a given value within the range of the pixel value groupand a pixel value of the attention pixel is larger than a value obtainedby multiplying the second index value by a second coefficient.
 6. Theimage processing device as defined in claim 5, the given value being amedian value among pixel values included in the pixel value group. 7.The image processing device as defined in claim 1, the isolated pointnoise correction section correcting the attention pixel using a valueobtained from the first to nth pixel values when it has been determinedthat the attention pixel is a pixel that corresponds to the isolatedpoint noise.
 8. The image processing device as defined in claim 7, theisolated point noise correction section correcting the attention pixelby replacing a pixel value of the attention pixel with a median valueamong the first to nth pixel values.
 9. The image processing device asdefined in claim 1, the image for which the isolated point noise iscorrected being an image in which pixels that differ in color aredisposed, and the first to nth pixel values being pixel values of pixelsamong the pixels included in the given area that are identical in colorwith the attention pixel.
 10. The image processing device as defined inclaim 1, further comprising: a highlight correction processing sectionthat determines whether or not the attention pixel that has beencorrected by the isolated point noise correction section is a pixelwithin a highlight area, and returns a pixel value of the attentionpixel to a pixel value before the attention pixel is corrected by theisolated point noise correction section when it has been determined thatthe attention pixel is a pixel within the highlight area.
 11. The imageprocessing device as defined in claim 10, the image for which theisolated point noise is corrected being an image in which pixels thatdiffer in color are disposed, and the highlight correction processingsection including an attention pixel highlight determination sectionthat compares the pixel value of the attention pixel that has not beencorrected by the isolated point noise correction section with pixelvalues of different-color adjacent pixels that differ in color from theattention pixel and are situated adjacent to the attention pixel todetermine whether or not the attention pixel is a candidate for a pixelwithin the highlight area.
 12. The image processing device as defined inclaim 11, the attention pixel highlight determination section comparinga pixel value of a pixel among the different-color adjacent pixels thathas not been corrected by the isolated point noise correction sectionwith the pixel value of the attention pixel that has not been correctedby the isolated point noise correction section.
 13. The image processingdevice as defined in claim 11, further comprising: a white balancecorrection section that performs a white balance correction process onthe different-color adjacent pixels, the attention pixel highlightdetermination section determining whether or not the attention pixel isa candidate for a pixel within the highlight area based on thedifferent-color adjacent pixels that have been subjected to the whitebalance correction process.
 14. The image processing device as definedin claim 11, the attention pixel highlight determination sectionselecting a pixel among the different-color adjacent pixels that has apixel value closest to the pixel value of the attention pixel as aselected pixel, and the highlight correction processing sectionincluding an adjacent pixel highlight determination section thatcompares the pixel value of the selected pixel with pixel values ofpixels that are identical in color with the selected pixel and aresituated at a given position with respect to the selected pixel todetermine whether or not the selected pixel is a pixel within thehighlight area, when the attention pixel highlight determination sectionhas determined that the attention pixel is a candidate for a pixelwithin the highlight area.
 15. The image processing device as defined inclaim 14, the highlight correction processing section including anattention pixel highlight decision section that determines that theattention pixel is a pixel within the highlight area when the attentionpixel highlight determination section has been determined that theattention pixel is a candidate for a pixel within the highlight area,and the adjacent pixel highlight determination section has determinedthat the different-color adjacent pixels are pixels within the highlightarea.
 16. An image processing device comprising: an isolated point noisedetection section that detects isolated point noise that is included inan image; and an isolated point noise correction section that correctsthe isolated point noise detected by the isolated point noise detectionsection, the isolated point noise detection section determining whetheror not the isolated point noise is included within a given areaincluding an attention pixel that is an isolated point noise detectiontarget, based on pixel values of pixels within the given area, anddetermining whether or not the attention pixel is a pixel thatcorresponds to the isolated point noise when it has been determined thatthe isolated point noise is included within the given area.
 17. Anendoscope apparatus comprising the image processing device as defined inclaim
 1. 18. An endoscope apparatus comprising the image processingdevice as defined in claim
 16. 19. An isolated point noise correctionmethod comprising: calculating a first index value that represents arange of first to nth pixel values, and a second index value thatrepresents a range of a pixel value group, the first to nth pixel valuesbeing obtained by arranging pixel values of pixels within a given areain ascending or descending order, the pixel value group being obtainedby excluding at least one of the first pixel value and the nth pixelvalue from the first to nth pixel values, and the given area includingan attention pixel that is an isolated point noise detection target;determining whether or not the isolated point noise is included withinthe given area based on the first index value and the second indexvalue; determining whether or not the attention pixel is a pixel thatcorresponds to the isolated point noise when it has been determined thatthe isolated point noise is included within the given area; andcorrecting a pixel value of the attention pixel when it has beendetermined that the attention pixel is a pixel that corresponds to theisolated point noise.
 20. A computer-readable storage device with anexecutable image processing program stored thereon, wherein the imageprocessing program instructs a computer to perform steps of: calculatinga first index value that represents a range of first to nth pixelvalues, and a second index value that represents a range of a pixelvalue group, the first to nth pixel values being obtained by arrangingpixel values of pixels within a given area in ascending or descendingorder, the pixel value group being obtained by excluding at least one ofthe first pixel value and the nth pixel value from the first to nthpixel values, and the given area including an attention pixel that is anisolated point noise detection target; determining whether or not theisolated point noise is included within the given area based on thefirst index value and the second index value; determining whether or notthe attention pixel is a pixel that corresponds to the isolated pointnoise when it has been determined that the isolated point noise isincluded within the given area; and correcting a pixel value of theattention pixel when it has been determined that the attention pixel isa pixel that corresponds to the isolated point noise.